SURVEY DEVELOPMENT
Questionnaire testing
Pre-testing
Pre-testing covers a range of testing techniques, the common feature being that the testing is conducted prior to taking a survey into the field (ie 'field test'). This phase of testing is critical for identifying problems for both respondents and interviewers, particularly regarding question content. The techniques used are designed to identify problems with the part of the questionnaire being tested and give some indication of the source of the error.
Techniques that were used during the pre-testing phase of questionnaire development:
- Focus groups—small groups of people who represent or who share similar characteristics with the target survey population. The ABS conducted NATSISS focus groups in September 2013 in remote locations in the Northern Territory. Participants were presented with proposed new and revised questions which were intended to be included in the survey and were asked to provide feedback about the proposed topics and question wording.
- Cognitive interviews—personal interviews conducted and recorded, with interviews including selected new and revised survey questions, and additional probing questions to establish if question wording is appropriate. Cognitive interviews were conducted in Queensland, Northern Territory, Victoria, South Australia and the ACT in September-October 2013.
- Expert evaluation—a peer review process used to identify respondent semantic problems, respondent task problems, assess content validity and translate concepts.
A major advantage of pre-testing is that small samples are used and limited survey specific documentation and training is required as the testing is performed by people working on the survey. Consequently the process can allow several iterative tests of a set of questions in the time it would take to conduct a pilot test.
The broad objectives of the 2014–15 NATSISS pre-testing were to:
- understand respondent interpretation and understanding of proposed concepts and questions;
- improve the construct validity of proposed questions; and
- ensure intended concepts were able to be captured.
Funding was received from the Department of the Prime Minister and Cabinet to enable the testing of new survey content relating to 'community strength and leadership'.
Field testing
The next phase of survey development involved field testing the survey questionnaire and procedures.
Dress rehearsal
The dress rehearsal is the final test in the development cycle and mainly focuses on the procedural and timing aspects of the survey. Primarily, it is an operational test. Questionnaire design errors (eg sequencing errors) can be identified, investigated and corrected. The objectives of the dress rehearsal were to:
- confirm the average interview time of the survey (for individual modules full survey);
- identify and rectify any issues that interviewers had with procedures, survey documentation, the survey questionnaire, or the Computer Assisted Interviewer Workload Management System;
- identify and rectify any issues that respondents had with survey content and structure;
- refine and add any necessary edits, edit notations or edit resolution instructions;
- refine and improve the survey's documentation (Interviewers' Instructions, Office Instructions, flowcharts, etc); and
- refine and improve interviewer training.
The dress rehearsal was conducted in March-April 2014 in both remote and non-remote locations in Queensland and Western Australia.
Print Page
Print All